The approaches described in this section could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
Computer software development often involves several progressive phases such as definition, coding, quality assurance (QA) and testing, error removal or “debugging,” and maintenance. In commercial software development, testing and debugging often consumes considerable time, and may account for the largest time component in a development project. Traditional bug fixing requires detailed communication between testers and developers. Time is often wasted going back and forth between testers and developers trying to reproduce and isolate problems. Verifying that a bug has been fixed is error-prone and time consuming. Therefore, software developers are continually looking for ways to reduce the amount of time spent on testing and debugging.
One approach is to outsource QA and testing to test personnel in another location, even another country. However, outsourcing can involve language barriers and communication barriers when a developer prepares and sends written bug reports in a local language to QA personnel in another country who speak another language. Distance barriers, and complications arising from having developers and testers in different physical locations or even different time zones also can exist. Developers would like to have a software testing solution that facilitates communication in an outsourced environment.
Scheduling QA testing within the software development cycle can be difficult. Finding and fixing bugs is unpredictable, at best. Delays in QA testing can lead to late releases, missed market windows, and lost revenue. These issues may be acute in the fast-paced environment involved in developing computer games for platforms such as the PC, Microsoft XBOX family, Sony Playstation family, Nintendo, etc.
To address these issues, several types of program execution recording systems have been developed. Generally, program-recording systems record information about a program under test as the program executes, and provide reports about that information. However, consistently reproducing bugs is a serious problem in software development, and achieving it typically involves expending significant resources. Some systems facilitate replaying program execution on a repeated basis. Using these systems, debugging is improved because bugs are reproducible by replaying a particular program execution session.
Types of program recording systems include external I/O recorders, source code instrumenters, and binary patching systems. Generally, external I/O recorders create and store recordings of network I/O operations, user input, and graphics or display 3-D (D3D) information. External I/O recorders may be use to simulate such external input to a program. External I/O recorders do not require any modifications to program code, are robust in response to code and asset changes, and can be used for cross-platform testing and demos. However, external I/O recorders may not produce accurate program replays due to timing issues, such as irregular network delays, or race conditions. Further, external I/O recorders may be unusable with some platforms for security reasons. External I/O recorders do not account for non-determinism in programs associated with thread context switching. External I/O recorders are sometimes termed macro recorders. A commercial example is Mercury Interactive's WinRunner.
Source code instrumenters usually provide proxy API libraries and modules; a developer must include the libraries or modules in source code for testing purposes. Source code instrumenters are somewhat reusable, can be easily expanded and tuned, and recordings may be portable across platforms. However, source code instrumenters are applicable only to modules for which program source code is available; they cannot be used to debug programs for which only executable machine code is available. Source code instrumenters typically require the use of a specialized API for certain calls by the developer, or a code-parsing module. Thus, the developer shoulders the burden of inserting the correct API calls in the source code of the program under test. Further, source code instrumenters provide no support for third-party modules such as dynamic linked libraries (DLLs) or linked executables, because such modules will not contain the required API calls at the time of testing. Source code instrumenters may not provide 100% accurate replays due to the effect of external events that are not trapped and recorded.
Binary patching systems operate by adding specialized recording code to the binary machine code of a program under test after compilation. A commercial example is Rational Purify. Binary patching systems are highly reusable and can produce accurate recordings by capturing detailed operational data. Binary patching systems do not require source code modifications, and can be applied to any executable, library or DLL. When disabled, binary patching systems do not affect program execution or size. Binary patching systems can capture low-level program calls, e.g., calls to hardware elements.
However, binary patching systems can be fragile when code or assets change. A recording of a program of a first version may be incompatible for replay when the program is modified to a later version. Binary patching systems may require special support for certain APIs, such as those relating to networking. Binary patching systems typically require special support for different processors having different machine instruction sets, and for different binary file formats (e.g., PE, XBE, ELF). Binary patching systems do not readily produce recordings that are portable across platforms. Further, cross-module inlining of code (e.g., using Link Time Code Generation (LTCG)) can distort function boundaries and make patching inaccurate.
In addition, known binary patching systems are not capable of recording all sources of non-determinism that may exist in an application.
Prior approaches have not provided efficient or convenient approaches for skipping ahead or backward to different points in execution of a program. One prior approach involves stack walking. Another prior approach, which is used for example in implementing “hibernate” functions of conventional personal computers, involves storing a copy of all values stored in memory—that is, the entire contents of memory—and elated state data on disk, and restoring the stored values when hibernation ends. While this approach captures the entire state of a system at a particular point in time, this approach is extremely inefficient because of the amount of data that needs to be stored and typically requires at least several seconds to accomplish a restore operation. Therefore, this approach is not practical for use when the state of the system needs to be stored frequently and restored rapidly. Further, the hibernation approach does not permit reverting or rewinding to a previous state of the system.